Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: cache pubkeys using base64 #7022

Closed
wants to merge 5 commits into from
Closed

feat: cache pubkeys using base64 #7022

wants to merge 5 commits into from

Conversation

twoeths
Copy link
Contributor

@twoeths twoeths commented Aug 14, 2024 •
edited
Loading

Motivation

Description

  • Use a single Buffer for toMemoryEfficientHexStr() function
  • switch to store pubkey as base64 instead of hex, this saves ~70MB for 2M validators as shown in the memory test. This is a good preparation for electra.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 14, 2024 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 98536e7 Previous: 20c18ad Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.0341 ms/op 1.8176 ms/op 1.12
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 62.655 us/op 40.225 us/op 1.56
BLS verify - blst 940.77 us/op 882.42 us/op 1.07
BLS verifyMultipleSignatures 3 - blst 1.3223 ms/op 1.3030 ms/op 1.01
BLS verifyMultipleSignatures 8 - blst 1.9581 ms/op 2.0903 ms/op 0.94
BLS verifyMultipleSignatures 32 - blst 5.7579 ms/op 4.4013 ms/op 1.31
BLS verifyMultipleSignatures 64 - blst 11.024 ms/op 8.2994 ms/op 1.33
BLS verifyMultipleSignatures 128 - blst 17.792 ms/op 15.802 ms/op 1.13
BLS deserializing 10000 signatures 707.53 ms/op 599.13 ms/op 1.18
BLS deserializing 100000 signatures 7.1146 s/op 6.1757 s/op 1.15
BLS verifyMultipleSignatures - same message - 3 - blst 922.54 us/op 998.52 us/op 0.92
BLS verifyMultipleSignatures - same message - 8 - blst 1.0842 ms/op 1.1268 ms/op 0.96
BLS verifyMultipleSignatures - same message - 32 - blst 1.7295 ms/op 1.6973 ms/op 1.02
BLS verifyMultipleSignatures - same message - 64 - blst 2.6434 ms/op 2.5039 ms/op 1.06
BLS verifyMultipleSignatures - same message - 128 - blst 4.4396 ms/op 4.2070 ms/op 1.06
BLS aggregatePubkeys 32 - blst 20.377 us/op 19.802 us/op 1.03
BLS aggregatePubkeys 128 - blst 72.570 us/op 64.850 us/op 1.12
notSeenSlots=1 numMissedVotes=1 numBadVotes=10 72.534 ms/op 67.537 ms/op 1.07
notSeenSlots=1 numMissedVotes=0 numBadVotes=4 59.113 ms/op 56.161 ms/op 1.05
notSeenSlots=2 numMissedVotes=1 numBadVotes=10 36.542 ms/op 31.897 ms/op 1.15
getSlashingsAndExits - default max 108.76 us/op 72.419 us/op 1.50
getSlashingsAndExits - 2k 342.34 us/op 278.27 us/op 1.23
proposeBlockBody type=full, size=empty 5.9669 ms/op 5.0896 ms/op 1.17
isKnown best case - 1 super set check 322.00 ns/op 464.00 ns/op 0.69
isKnown normal case - 2 super set checks 313.00 ns/op 458.00 ns/op 0.68
isKnown worse case - 16 super set checks 324.00 ns/op 452.00 ns/op 0.72
InMemoryCheckpointStateCache - add get delete 3.2060 us/op 2.6270 us/op 1.22
validate api signedAggregateAndProof - struct 1.4938 ms/op 1.9772 ms/op 0.76
validate gossip signedAggregateAndProof - struct 1.5064 ms/op 1.9286 ms/op 0.78
validate gossip attestation - vc 640000 1.0171 ms/op 995.59 us/op 1.02
batch validate gossip attestation - vc 640000 - chunk 32 138.50 us/op 120.55 us/op 1.15
batch validate gossip attestation - vc 640000 - chunk 64 117.52 us/op 104.29 us/op 1.13
batch validate gossip attestation - vc 640000 - chunk 128 106.13 us/op 99.233 us/op 1.07
batch validate gossip attestation - vc 640000 - chunk 256 108.16 us/op 91.738 us/op 1.18
pickEth1Vote - no votes 1.0448 ms/op 856.03 us/op 1.22
pickEth1Vote - max votes 5.5271 ms/op 5.8577 ms/op 0.94
pickEth1Vote - Eth1Data hashTreeRoot value x2048 11.944 ms/op 14.583 ms/op 0.82
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 17.711 ms/op 19.847 ms/op 0.89
pickEth1Vote - Eth1Data fastSerialize value x2048 467.67 us/op 379.25 us/op 1.23
pickEth1Vote - Eth1Data fastSerialize tree x2048 2.3780 ms/op 3.8007 ms/op 0.63
bytes32 toHexString 416.00 ns/op 623.00 ns/op 0.67
bytes32 Buffer.toString(hex) 248.00 ns/op 455.00 ns/op 0.55
bytes32 Buffer.toString(hex) from Uint8Array 352.00 ns/op 643.00 ns/op 0.55
bytes32 Buffer.toString(hex) + 0x 246.00 ns/op 469.00 ns/op 0.52
Object access 1 prop 0.14200 ns/op 0.32800 ns/op 0.43
Map access 1 prop 0.13200 ns/op 0.36200 ns/op 0.36
Object get x1000 5.8820 ns/op 5.5030 ns/op 1.07
Map get x1000 6.4290 ns/op 5.9860 ns/op 1.07
Object set x1000 32.512 ns/op 23.827 ns/op 1.36
Map set x1000 22.628 ns/op 19.903 ns/op 1.14
Return object 10000 times 0.29210 ns/op 0.30000 ns/op 0.97
Throw Error 10000 times 3.3566 us/op 2.8146 us/op 1.19
toHex 136.58 ns/op 112.51 ns/op 1.21
Buffer.from 138.18 ns/op 104.45 ns/op 1.32
shared Buffer 86.021 ns/op 71.059 ns/op 1.21
fastMsgIdFn sha256 / 200 bytes 2.1780 us/op 1.9710 us/op 1.11
fastMsgIdFn h32 xxhash / 200 bytes 225.00 ns/op 449.00 ns/op 0.50
fastMsgIdFn h64 xxhash / 200 bytes 274.00 ns/op 468.00 ns/op 0.59
fastMsgIdFn sha256 / 1000 bytes 7.2340 us/op 5.9190 us/op 1.22
fastMsgIdFn h32 xxhash / 1000 bytes 354.00 ns/op 573.00 ns/op 0.62
fastMsgIdFn h64 xxhash / 1000 bytes 349.00 ns/op 535.00 ns/op 0.65
fastMsgIdFn sha256 / 10000 bytes 64.538 us/op 50.664 us/op 1.27
fastMsgIdFn h32 xxhash / 10000 bytes 1.8580 us/op 1.9870 us/op 0.94
fastMsgIdFn h64 xxhash / 10000 bytes 1.2250 us/op 1.3820 us/op 0.89
send data - 1000 256B messages 11.956 ms/op 9.6131 ms/op 1.24
send data - 1000 512B messages 16.371 ms/op 14.072 ms/op 1.16
send data - 1000 1024B messages 26.679 ms/op 22.236 ms/op 1.20
send data - 1000 1200B messages 26.074 ms/op 21.958 ms/op 1.19
send data - 1000 2048B messages 32.921 ms/op 30.813 ms/op 1.07
send data - 1000 4096B messages 32.075 ms/op 27.409 ms/op 1.17
send data - 1000 16384B messages 71.813 ms/op 72.365 ms/op 0.99
send data - 1000 65536B messages 214.98 ms/op 288.78 ms/op 0.74
enrSubnets - fastDeserialize 64 bits 1.0630 us/op 1.2920 us/op 0.82
enrSubnets - ssz BitVector 64 bits 362.00 ns/op 577.00 ns/op 0.63
enrSubnets - fastDeserialize 4 bits 154.00 ns/op 350.00 ns/op 0.44
enrSubnets - ssz BitVector 4 bits 360.00 ns/op 557.00 ns/op 0.65
prioritizePeers score -10:0 att 32-0.1 sync 2-0 147.28 us/op 134.93 us/op 1.09
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 169.61 us/op 132.57 us/op 1.28
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 286.55 us/op 207.04 us/op 1.38
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 469.87 us/op 407.99 us/op 1.15
prioritizePeers score 0:0 att 64-1 sync 4-1 605.41 us/op 523.57 us/op 1.16
array of 16000 items push then shift 1.6452 us/op 1.2955 us/op 1.27
LinkedList of 16000 items push then shift 7.4600 ns/op 6.4490 ns/op 1.16
array of 16000 items push then pop 113.23 ns/op 102.61 ns/op 1.10
LinkedList of 16000 items push then pop 7.2890 ns/op 6.3220 ns/op 1.15
array of 24000 items push then shift 2.5167 us/op 1.8808 us/op 1.34
LinkedList of 24000 items push then shift 7.3430 ns/op 6.4040 ns/op 1.15
array of 24000 items push then pop 155.26 ns/op 111.06 ns/op 1.40
LinkedList of 24000 items push then pop 7.2750 ns/op 6.1930 ns/op 1.17
intersect bitArray bitLen 8 6.7590 ns/op 5.3680 ns/op 1.26
intersect array and set length 8 48.043 ns/op 39.347 ns/op 1.22
intersect bitArray bitLen 128 31.312 ns/op 26.139 ns/op 1.20
intersect array and set length 128 698.05 ns/op 582.76 ns/op 1.20
bitArray.getTrueBitIndexes() bitLen 128 1.9420 us/op 1.9440 us/op 1.00
bitArray.getTrueBitIndexes() bitLen 248 3.8540 us/op 3.2020 us/op 1.20
bitArray.getTrueBitIndexes() bitLen 512 9.6510 us/op 5.5860 us/op 1.73
Buffer.concat 32 items 1000.0 ns/op 1.1300 us/op 0.88
Uint8Array.set 32 items 1.6900 us/op 1.4380 us/op 1.18
Buffer.copy 2.2230 us/op 2.5710 us/op 0.86
Uint8Array.set - with subarray 3.3950 us/op 3.6650 us/op 0.93
Uint8Array.set - without subarray 1.8540 us/op 2.0500 us/op 0.90
getUint32 - dataview 307.00 ns/op 411.00 ns/op 0.75
getUint32 - manual 244.00 ns/op 342.00 ns/op 0.71
Set add up to 64 items then delete first 2.5410 us/op 1.8255 us/op 1.39
OrderedSet add up to 64 items then delete first 3.9764 us/op 2.8260 us/op 1.41
Set add up to 64 items then delete last 3.2842 us/op 2.0983 us/op 1.57
OrderedSet add up to 64 items then delete last 4.9475 us/op 3.0972 us/op 1.60
Set add up to 64 items then delete middle 3.4341 us/op 2.0547 us/op 1.67
OrderedSet add up to 64 items then delete middle 6.0622 us/op 4.4751 us/op 1.35
Set add up to 128 items then delete first 6.6238 us/op 4.0743 us/op 1.63
OrderedSet add up to 128 items then delete first 8.9463 us/op 6.3079 us/op 1.42
Set add up to 128 items then delete last 5.8002 us/op 3.9434 us/op 1.47
OrderedSet add up to 128 items then delete last 7.9008 us/op 5.9308 us/op 1.33
Set add up to 128 items then delete middle 5.2078 us/op 3.9992 us/op 1.30
OrderedSet add up to 128 items then delete middle 15.090 us/op 12.103 us/op 1.25
Set add up to 256 items then delete first 11.055 us/op 8.0880 us/op 1.37
OrderedSet add up to 256 items then delete first 17.303 us/op 12.659 us/op 1.37
Set add up to 256 items then delete last 10.621 us/op 7.7839 us/op 1.36
OrderedSet add up to 256 items then delete last 15.998 us/op 11.803 us/op 1.36
Set add up to 256 items then delete middle 10.675 us/op 7.6456 us/op 1.40
OrderedSet add up to 256 items then delete middle 46.545 us/op 34.147 us/op 1.36
transfer serialized Status (84 B) 1.5100 us/op 1.5400 us/op 0.98
copy serialized Status (84 B) 1.3330 us/op 1.3270 us/op 1.00
transfer serialized SignedVoluntaryExit (112 B) 1.6640 us/op 1.7750 us/op 0.94
copy serialized SignedVoluntaryExit (112 B) 1.4270 us/op 1.6430 us/op 0.87
transfer serialized ProposerSlashing (416 B) 2.6280 us/op 2.8140 us/op 0.93
copy serialized ProposerSlashing (416 B) 2.8710 us/op 2.9380 us/op 0.98
transfer serialized Attestation (485 B) 2.7580 us/op 2.7590 us/op 1.00
copy serialized Attestation (485 B) 2.8700 us/op 2.4210 us/op 1.19
transfer serialized AttesterSlashing (33232 B) 2.9840 us/op 2.5260 us/op 1.18
copy serialized AttesterSlashing (33232 B) 7.9340 us/op 4.9890 us/op 1.59
transfer serialized Small SignedBeaconBlock (128000 B) 2.5450 us/op 3.2880 us/op 0.77
copy serialized Small SignedBeaconBlock (128000 B) 15.967 us/op 10.956 us/op 1.46
transfer serialized Avg SignedBeaconBlock (200000 B) 3.0620 us/op 3.0430 us/op 1.01
copy serialized Avg SignedBeaconBlock (200000 B) 22.301 us/op 15.776 us/op 1.41
transfer serialized BlobsSidecar (524380 B) 2.7970 us/op 3.7470 us/op 0.75
copy serialized BlobsSidecar (524380 B) 191.81 us/op 74.591 us/op 2.57
transfer serialized Big SignedBeaconBlock (1000000 B) 2.9160 us/op 3.5880 us/op 0.81
copy serialized Big SignedBeaconBlock (1000000 B) 152.66 us/op 186.03 us/op 0.82
pass gossip attestations to forkchoice per slot 2.9687 ms/op 2.4717 ms/op 1.20
forkChoice updateHead vc 100000 bc 64 eq 0 531.39 us/op 431.87 us/op 1.23
forkChoice updateHead vc 600000 bc 64 eq 0 4.2751 ms/op 2.6716 ms/op 1.60
forkChoice updateHead vc 1000000 bc 64 eq 0 5.6983 ms/op 4.2338 ms/op 1.35
forkChoice updateHead vc 600000 bc 320 eq 0 3.4277 ms/op 2.5821 ms/op 1.33
forkChoice updateHead vc 600000 bc 1200 eq 0 3.6205 ms/op 2.7021 ms/op 1.34
forkChoice updateHead vc 600000 bc 7200 eq 0 4.5461 ms/op 3.2161 ms/op 1.41
forkChoice updateHead vc 600000 bc 64 eq 1000 11.438 ms/op 9.7068 ms/op 1.18
forkChoice updateHead vc 600000 bc 64 eq 10000 11.670 ms/op 9.5604 ms/op 1.22
forkChoice updateHead vc 600000 bc 64 eq 300000 16.165 ms/op 11.835 ms/op 1.37
computeDeltas 500000 validators 300 proto nodes 5.6664 ms/op 3.0339 ms/op 1.87
computeDeltas 500000 validators 1200 proto nodes 5.5084 ms/op 3.2527 ms/op 1.69
computeDeltas 500000 validators 7200 proto nodes 5.2212 ms/op 3.2553 ms/op 1.60
computeDeltas 750000 validators 300 proto nodes 8.2797 ms/op 4.7079 ms/op 1.76
computeDeltas 750000 validators 1200 proto nodes 7.5950 ms/op 4.6277 ms/op 1.64
computeDeltas 750000 validators 7200 proto nodes 7.6541 ms/op 4.6213 ms/op 1.66
computeDeltas 1400000 validators 300 proto nodes 12.606 ms/op 8.2808 ms/op 1.52
computeDeltas 1400000 validators 1200 proto nodes 12.648 ms/op 8.6303 ms/op 1.47
computeDeltas 1400000 validators 7200 proto nodes 13.154 ms/op 8.4612 ms/op 1.55
computeDeltas 2100000 validators 300 proto nodes 18.086 ms/op 12.383 ms/op 1.46
computeDeltas 2100000 validators 1200 proto nodes 20.322 ms/op 12.574 ms/op 1.62
computeDeltas 2100000 validators 7200 proto nodes 23.043 ms/op 12.701 ms/op 1.81
altair processAttestation - 250000 vs - 7PWei normalcase 1.6972 ms/op 1.4097 ms/op 1.20
altair processAttestation - 250000 vs - 7PWei worstcase 2.5071 ms/op 2.1595 ms/op 1.16
altair processAttestation - setStatus - 1/6 committees join 86.085 us/op 70.380 us/op 1.22
altair processAttestation - setStatus - 1/3 committees join 173.87 us/op 150.64 us/op 1.15
altair processAttestation - setStatus - 1/2 committees join 252.70 us/op 214.73 us/op 1.18
altair processAttestation - setStatus - 2/3 committees join 329.16 us/op 273.34 us/op 1.20
altair processAttestation - setStatus - 4/5 committees join 476.77 us/op 395.84 us/op 1.20
altair processAttestation - setStatus - 100% committees join 554.39 us/op 473.67 us/op 1.17
altair processBlock - 250000 vs - 7PWei normalcase 4.1409 ms/op 4.1178 ms/op 1.01
altair processBlock - 250000 vs - 7PWei normalcase hashState 24.489 ms/op 27.553 ms/op 0.89
altair processBlock - 250000 vs - 7PWei worstcase 39.077 ms/op 39.538 ms/op 0.99
altair processBlock - 250000 vs - 7PWei worstcase hashState 78.936 ms/op 74.982 ms/op 1.05
phase0 processBlock - 250000 vs - 7PWei normalcase 2.3088 ms/op 2.5842 ms/op 0.89
phase0 processBlock - 250000 vs - 7PWei worstcase 25.685 ms/op 25.861 ms/op 0.99
altair processEth1Data - 250000 vs - 7PWei normalcase 288.87 us/op 253.55 us/op 1.14
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 7.5340 us/op 4.2250 us/op 1.78
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 30.494 us/op 22.396 us/op 1.36
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 11.337 us/op 9.3330 us/op 1.21
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 8.6140 us/op 6.9700 us/op 1.24
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 114.04 us/op 73.743 us/op 1.55
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 1.1783 ms/op 578.77 us/op 2.04
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 1.5605 ms/op 863.18 us/op 1.81
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 1.6513 ms/op 726.62 us/op 2.27
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 2.5058 ms/op 2.3107 ms/op 1.08
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 1.6310 ms/op 1.1897 ms/op 1.37
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 3.7185 ms/op 3.3296 ms/op 1.12
Public key to string using hex, shared Buffer instance 108.07 ns/op
Public key to string using hex, separate Buffer instance 157.07 ns/op
Public key to string using base64, shared Buffer instance 89.193 ns/op
Public key to string using base64, separate Buffer instance 151.75 ns/op
Tree 40 250000 create 208.05 ms/op 179.01 ms/op 1.16
Tree 40 250000 get(125000) 142.47 ns/op 115.12 ns/op 1.24
Tree 40 250000 set(125000) 667.40 ns/op 543.76 ns/op 1.23
Tree 40 250000 toArray() 15.596 ms/op 11.279 ms/op 1.38
Tree 40 250000 iterate all - toArray() + loop 18.030 ms/op 16.708 ms/op 1.08
Tree 40 250000 iterate all - get(i) 54.194 ms/op 44.622 ms/op 1.21
MutableVector 250000 create 8.7768 ms/op 11.756 ms/op 0.75
MutableVector 250000 get(125000) 6.5530 ns/op 5.6290 ns/op 1.16
MutableVector 250000 set(125000) 214.80 ns/op 183.90 ns/op 1.17
MutableVector 250000 toArray() 3.4429 ms/op 3.4570 ms/op 1.00
MutableVector 250000 iterate all - toArray() + loop 3.6005 ms/op 3.6338 ms/op 0.99
MutableVector 250000 iterate all - get(i) 1.5490 ms/op 1.3878 ms/op 1.12
Array 250000 create 3.1517 ms/op 3.2475 ms/op 0.97
Array 250000 clone - spread 1.5057 ms/op 1.3461 ms/op 1.12
Array 250000 get(125000) 0.41600 ns/op 0.55600 ns/op 0.75
Array 250000 set(125000) 0.43800 ns/op 0.56700 ns/op 0.77
Array 250000 iterate all - loop 105.11 us/op 78.628 us/op 1.34
effectiveBalanceIncrements clone Uint8Array 300000 31.017 us/op 17.002 us/op 1.82
effectiveBalanceIncrements clone MutableVector 300000 129.00 ns/op 313.00 ns/op 0.41
effectiveBalanceIncrements rw all Uint8Array 300000 198.47 us/op 166.35 us/op 1.19
effectiveBalanceIncrements rw all MutableVector 300000 69.511 ms/op 61.039 ms/op 1.14
phase0 afterProcessEpoch - 250000 vs - 7PWei 94.502 ms/op 77.672 ms/op 1.22
Array.fill - length 1000000 3.7144 ms/op 2.7915 ms/op 1.33
Array push - length 1000000 15.933 ms/op 14.964 ms/op 1.06
Array.get 0.29457 ns/op 0.25873 ns/op 1.14
Uint8Array.get 0.44842 ns/op 0.33315 ns/op 1.35
phase0 beforeProcessEpoch - 250000 vs - 7PWei 16.673 ms/op 19.209 ms/op 0.87
altair processEpoch - mainnet_e81889 337.08 ms/op 303.99 ms/op 1.11
mainnet_e81889 - altair beforeProcessEpoch 17.513 ms/op 20.380 ms/op 0.86
mainnet_e81889 - altair processJustificationAndFinalization 11.874 us/op 13.417 us/op 0.88
mainnet_e81889 - altair processInactivityUpdates 5.4809 ms/op 5.7530 ms/op 0.95
mainnet_e81889 - altair processRewardsAndPenalties 59.347 ms/op 50.289 ms/op 1.18
mainnet_e81889 - altair processRegistryUpdates 2.0180 us/op 2.3950 us/op 0.84
mainnet_e81889 - altair processSlashings 471.00 ns/op 977.00 ns/op 0.48
mainnet_e81889 - altair processEth1DataReset 306.00 ns/op 765.00 ns/op 0.40
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.3325 ms/op 877.50 us/op 1.52
mainnet_e81889 - altair processSlashingsReset 4.9360 us/op 2.6000 us/op 1.90
mainnet_e81889 - altair processRandaoMixesReset 5.4490 us/op 6.6790 us/op 0.82
mainnet_e81889 - altair processHistoricalRootsUpdate 625.00 ns/op 984.00 ns/op 0.64
mainnet_e81889 - altair processParticipationFlagUpdates 2.4640 us/op 3.2970 us/op 0.75
mainnet_e81889 - altair processSyncCommitteeUpdates 460.00 ns/op 975.00 ns/op 0.47
mainnet_e81889 - altair afterProcessEpoch 101.99 ms/op 78.820 ms/op 1.29
capella processEpoch - mainnet_e217614 1.3441 s/op 1.2454 s/op 1.08
mainnet_e217614 - capella beforeProcessEpoch 74.540 ms/op 71.270 ms/op 1.05
mainnet_e217614 - capella processJustificationAndFinalization 18.442 us/op 20.158 us/op 0.91
mainnet_e217614 - capella processInactivityUpdates 16.819 ms/op 12.839 ms/op 1.31
mainnet_e217614 - capella processRewardsAndPenalties 255.60 ms/op 251.90 ms/op 1.01
mainnet_e217614 - capella processRegistryUpdates 18.716 us/op 16.558 us/op 1.13
mainnet_e217614 - capella processSlashings 521.00 ns/op 905.00 ns/op 0.58
mainnet_e217614 - capella processEth1DataReset 587.00 ns/op 983.00 ns/op 0.60
mainnet_e217614 - capella processEffectiveBalanceUpdates 10.110 ms/op 11.878 ms/op 0.85
mainnet_e217614 - capella processSlashingsReset 2.7300 us/op 6.0470 us/op 0.45
mainnet_e217614 - capella processRandaoMixesReset 6.4400 us/op 7.1890 us/op 0.90
mainnet_e217614 - capella processHistoricalRootsUpdate 848.00 ns/op 1.8630 us/op 0.46
mainnet_e217614 - capella processParticipationFlagUpdates 1.9140 us/op 5.4320 us/op 0.35
mainnet_e217614 - capella afterProcessEpoch 242.53 ms/op 314.19 ms/op 0.77
phase0 processEpoch - mainnet_e58758 412.61 ms/op 472.64 ms/op 0.87
mainnet_e58758 - phase0 beforeProcessEpoch 91.580 ms/op 127.19 ms/op 0.72
mainnet_e58758 - phase0 processJustificationAndFinalization 23.897 us/op 32.478 us/op 0.74
mainnet_e58758 - phase0 processRewardsAndPenalties 42.185 ms/op 42.717 ms/op 0.99
mainnet_e58758 - phase0 processRegistryUpdates 12.018 us/op 14.369 us/op 0.84
mainnet_e58758 - phase0 processSlashings 440.00 ns/op 1.1070 us/op 0.40
mainnet_e58758 - phase0 processEth1DataReset 394.00 ns/op 1.2580 us/op 0.31
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.7556 ms/op 1.3288 ms/op 1.32
mainnet_e58758 - phase0 processSlashingsReset 5.1700 us/op 5.6440 us/op 0.92
mainnet_e58758 - phase0 processRandaoMixesReset 6.1620 us/op 9.6410 us/op 0.64
mainnet_e58758 - phase0 processHistoricalRootsUpdate 484.00 ns/op 1.0820 us/op 0.45
mainnet_e58758 - phase0 processParticipationRecordUpdates 4.4710 us/op 9.7610 us/op 0.46
mainnet_e58758 - phase0 afterProcessEpoch 82.336 ms/op 96.441 ms/op 0.85
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.5644 ms/op 1.6814 ms/op 0.93
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.7529 ms/op 2.3713 ms/op 0.74
altair processInactivityUpdates - 250000 normalcase 21.399 ms/op 23.520 ms/op 0.91
altair processInactivityUpdates - 250000 worstcase 20.038 ms/op 22.529 ms/op 0.89
phase0 processRegistryUpdates - 250000 normalcase 9.3340 us/op 13.760 us/op 0.68
phase0 processRegistryUpdates - 250000 badcase_full_deposits 280.34 us/op 368.14 us/op 0.76
phase0 processRegistryUpdates - 250000 worstcase 0.5 131.25 ms/op 192.50 ms/op 0.68
altair processRewardsAndPenalties - 250000 normalcase 44.961 ms/op 63.725 ms/op 0.71
altair processRewardsAndPenalties - 250000 worstcase 42.295 ms/op 56.079 ms/op 0.75
phase0 getAttestationDeltas - 250000 normalcase 7.6728 ms/op 11.228 ms/op 0.68
phase0 getAttestationDeltas - 250000 worstcase 7.4544 ms/op 7.7475 ms/op 0.96
phase0 processSlashings - 250000 worstcase 119.80 us/op 93.152 us/op 1.29
altair processSyncCommitteeUpdates - 250000 123.58 ms/op 118.24 ms/op 1.05
BeaconState.hashTreeRoot - No change 286.00 ns/op 705.00 ns/op 0.41
BeaconState.hashTreeRoot - 1 full validator 126.72 us/op 121.75 us/op 1.04
BeaconState.hashTreeRoot - 32 full validator 837.93 us/op 1.4721 ms/op 0.57
BeaconState.hashTreeRoot - 512 full validator 9.9168 ms/op 11.412 ms/op 0.87
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 103.22 us/op 158.17 us/op 0.65
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.4593 ms/op 1.8565 ms/op 0.79
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 24.017 ms/op 29.440 ms/op 0.82
BeaconState.hashTreeRoot - 1 balances 122.06 us/op 124.94 us/op 0.98
BeaconState.hashTreeRoot - 32 balances 759.59 us/op 1.4404 ms/op 0.53
BeaconState.hashTreeRoot - 512 balances 8.4729 ms/op 9.8619 ms/op 0.86
BeaconState.hashTreeRoot - 250000 balances 151.88 ms/op 248.71 ms/op 0.61
aggregationBits - 2048 els - zipIndexesInBitList 23.528 us/op 37.214 us/op 0.63
byteArrayEquals 32 53.716 ns/op 53.345 ns/op 1.01
Buffer.compare 32 17.060 ns/op 16.785 ns/op 1.02
byteArrayEquals 1024 1.5736 us/op 1.2965 us/op 1.21
Buffer.compare 1024 25.400 ns/op 25.098 ns/op 1.01
byteArrayEquals 16384 24.954 us/op 20.481 us/op 1.22
Buffer.compare 16384 190.67 ns/op 197.46 ns/op 0.97
byteArrayEquals 123687377 190.28 ms/op 160.15 ms/op 1.19
Buffer.compare 123687377 6.6725 ms/op 6.1955 ms/op 1.08
byteArrayEquals 32 - diff last byte 52.358 ns/op 47.800 ns/op 1.10
Buffer.compare 32 - diff last byte 17.129 ns/op 16.810 ns/op 1.02
byteArrayEquals 1024 - diff last byte 1.5754 us/op 1.2838 us/op 1.23
Buffer.compare 1024 - diff last byte 24.991 ns/op 26.563 ns/op 0.94
byteArrayEquals 16384 - diff last byte 25.202 us/op 20.241 us/op 1.25
Buffer.compare 16384 - diff last byte 200.57 ns/op 211.35 ns/op 0.95
byteArrayEquals 123687377 - diff last byte 188.94 ms/op 154.47 ms/op 1.22
Buffer.compare 123687377 - diff last byte 6.2011 ms/op 5.2619 ms/op 1.18
byteArrayEquals 32 - random bytes 7.5970 ns/op 4.9930 ns/op 1.52
Buffer.compare 32 - random bytes 17.153 ns/op 16.051 ns/op 1.07
byteArrayEquals 1024 - random bytes 5.0620 ns/op 4.9610 ns/op 1.02
Buffer.compare 1024 - random bytes 16.907 ns/op 16.107 ns/op 1.05
byteArrayEquals 16384 - random bytes 7.6880 ns/op 4.9560 ns/op 1.55
Buffer.compare 16384 - random bytes 16.905 ns/op 15.742 ns/op 1.07
byteArrayEquals 123687377 - random bytes 8.9200 ns/op 7.7200 ns/op 1.16
Buffer.compare 123687377 - random bytes 18.170 ns/op 19.770 ns/op 0.92
regular array get 100000 times 43.364 us/op 31.152 us/op 1.39
wrappedArray get 100000 times 32.525 us/op 31.045 us/op 1.05
arrayWithProxy get 100000 times 14.163 ms/op 9.7454 ms/op 1.45
ssz.Root.equals 45.487 ns/op 43.086 ns/op 1.06
byteArrayEquals 44.893 ns/op 42.806 ns/op 1.05
Buffer.compare 10.210 ns/op 9.1160 ns/op 1.12
shuffle list - 16384 els 6.1827 ms/op 5.5635 ms/op 1.11
shuffle list - 250000 els 92.194 ms/op 82.024 ms/op 1.12
processSlot - 1 slots 12.020 us/op 11.763 us/op 1.02
processSlot - 32 slots 3.4858 ms/op 3.2720 ms/op 1.07
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 42.528 ms/op 37.073 ms/op 1.15
getCommitteeAssignments - req 1 vs - 250000 vc 2.1449 ms/op 1.7576 ms/op 1.22
getCommitteeAssignments - req 100 vs - 250000 vc 4.2069 ms/op 3.5043 ms/op 1.20
getCommitteeAssignments - req 1000 vs - 250000 vc 4.3113 ms/op 3.7736 ms/op 1.14
findModifiedValidators - 10000 modified validators 236.40 ms/op 229.70 ms/op 1.03
findModifiedValidators - 1000 modified validators 161.17 ms/op 139.75 ms/op 1.15
findModifiedValidators - 100 modified validators 151.38 ms/op 146.43 ms/op 1.03
findModifiedValidators - 10 modified validators 149.28 ms/op 134.33 ms/op 1.11
findModifiedValidators - 1 modified validators 153.72 ms/op 138.11 ms/op 1.11
findModifiedValidators - no difference 152.97 ms/op 153.75 ms/op 0.99
compare ViewDUs 3.3076 s/op 3.0420 s/op 1.09
compare each validator Uint8Array 1.3819 s/op 1.4135 s/op 0.98
compare ViewDU to Uint8Array 1.2921 s/op 769.88 ms/op 1.68
migrate state 1000000 validators, 24 modified, 0 new 704.25 ms/op 508.02 ms/op 1.39
migrate state 1000000 validators, 1700 modified, 1000 new 1.0137 s/op 797.79 ms/op 1.27
migrate state 1000000 validators, 3400 modified, 2000 new 1.2011 s/op 1.0984 s/op 1.09
migrate state 1500000 validators, 24 modified, 0 new 734.46 ms/op 640.35 ms/op 1.15
migrate state 1500000 validators, 1700 modified, 1000 new 961.05 ms/op 828.75 ms/op 1.16
migrate state 1500000 validators, 3400 modified, 2000 new 1.1416 s/op 1.1481 s/op 0.99
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.7300 ns/op 6.8800 ns/op 0.69
state getBlockRootAtSlot - 250000 vs - 7PWei 485.90 ns/op 908.60 ns/op 0.53
computeProposers - vc 250000 7.9305 ms/op 6.5108 ms/op 1.22
computeEpochShuffling - vc 250000 99.869 ms/op 84.922 ms/op 1.18
getNextSyncCommittee - vc 250000 130.36 ms/op 124.43 ms/op 1.05
computeSigningRoot for AttestationData 20.694 us/op 19.859 us/op 1.04
hash AttestationData serialized data then Buffer.toString(base64) 1.6528 us/op 1.2247 us/op 1.35
toHexString serialized data 914.14 ns/op 932.04 ns/op 0.98
Buffer.toString(base64) 186.34 ns/op 193.59 ns/op 0.96
block root to RootHex using toHex 164.82 ns/op 139.49 ns/op 1.18
block root to RootHex using toRootHex 99.950 ns/op 88.903 ns/op 1.12

by benchmarkbot/action

@codecov Codecov
Copy link

codecov bot commented Aug 14, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 91.30435% with 2 lines in your changes missing coverage. Please review.

Project coverage is 49.27%. Comparing base (44b2156) to head (02aa7b8).
Report is 119 commits behind head on unstable.

Additional details and impacted files 
@@             Coverage Diff              @@
##           unstable    #7022      +/-   ##
============================================
+ Coverage     49.24%   49.27%   +0.02%     
============================================
  Files           578      578              
  Lines         37438    37456      +18     
  Branches       2162     2176      +14     
============================================
+ Hits          18435    18455      +20     
+ Misses        18963    18960       -3     
- Partials         40       41       +1

@twoeths twoeths marked this pull request as ready for review August 14, 2024 06:44
@twoeths twoeths requested a review from a team as a code owner August 14, 2024 06:44
@twoeths twoeths marked this pull request as draft August 14, 2024 07:03
@twoeths
Copy link
Contributor Author

twoeths commented Aug 14, 2024

converting this to Draft, I think using base64 is 33% better than hex in terms of memory allocation, which is the main concern of this function as noted in #3039 so we should try it

@twoeths twoeths changed the title fix: use shared Buffer for toMemoryEfficientHexStr() feat: cache pubkeys using base64 Aug 15, 2024
@twoeths twoeths marked this pull request as ready for review August 15, 2024 09:41
@wemeetagain
Copy link
Member

alternative is to try https://github.com/ChainSafe/pubkey-index-map/

@philknows philknows added this to the v1.22.0 milestone Sep 4, 2024
@philknows philknows modified the milestones: v1.22.0, v1.23.0 Sep 11, 2024
@twoeths twoeths mentioned this pull request Sep 18, 2024
Merged
@twoeths twoeths marked this pull request as draft September 18, 2024 08:20
@twoeths
Copy link
Contributor Author

twoeths commented Sep 27, 2024

will go with #7091

@twoeths twoeths closed this Sep 27, 2024
@twoeths twoeths deleted the te/shared_buffer_sync_pubkeys branch September 27, 2024 02:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v1.23.0
Development

Successfully merging this pull request may close these issues.
3 participants
@twoeths @wemeetagain @philknows